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ABSTRACT 

This study highlights the value of leveraging department resources in principal 
preparation program resources to achieve multiple teaching and program review goals. 
In this study masters degree candidates participated in all aspects of the evaluation of 
the Department of Educational Administration and Counseling. As consumers they 
already had a sense of what issues would be the most relevant to examine. As learners 
they participated in every aspect of program review. The openness of the Department 
modeled the kind of openness future administrators will have to exhibit in their work. 
The Department was able to take the findings of the study to develop a master plan for 
program improvement. 



T he ability to identify essential information and organize it in a meaningful way is fast 
becoming one of the most important skills for leaders. Maruska (2004) argues that in 
order to improve an organization, the leader must seek better information that can 
inform a hopeful future. Kotter (1996) warned that without an appropriate [informed] 
vision, a transformation effort can easily dissolve into a list of confusing, incompatible, and 
time-consuming projects that go nowhere” (p. 7). Peters (2003) underscored the urgency for 
leaders to understand as much about their own organizations as their customers do: “. . . We 
Live in the Age of the Never Satisfied Customer ... [Information Technology] enables total 
transparency. People with access to 
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relevant information are beginning to challenge any kind of authority. The stupid, loyal and 
humble customer, employee, and citizen is dead” (p. 67). 

For any organization to formulate its mission, understand its purpose and be credible 
with its various audiences, one of the most essential functions for a leader is that of creatively 
scanning the environment for signs of opportunities and dangers. This concept is not new. 
Worthen & Sanders (1987) argued that educational systems have a key deficit when they do 
not systematically evaluate their processes. “Without careful, systematic inquiry into the 
effectiveness of either current school practice or new programs, many changes occurring in 
education become little more than random adoption of faddish innovations” (pp. 3-4). 

Accordingly, one the most important competencies of the 21 st century school 
administrator is the ability to identify barriers to learning and take action to build a better 
learning environment. What makes systematic data gathering and analysis more urgent is the 
fact that information is now in the hands of schools various constituencies - parents, students 
and teachers. This in turn is leading to unprecedented demands for accountability and 
improved quality of service. Ultimately, school leaders who do not take program evaluation 
seriously will fall victim to an increasingly informed and sophisticated clientele who will 
select and use data in their own interest. 

How to ensure the development of information gathering and analysis skills of 
current and future principals is the challenge of principal preparation programs across the 
nation. Principals who do not learn these skills risk losing credibility. Their ability to control 
schools will go to others who are better at analyzing and interpreting data. We know that 
administrators do not learn these skills from a classroom lecture, a textbook, or even a 
weekend retreat with a dynamic speaker and lots of good food. Therefore, it is emphatic that 
principal preparation programs ensure that their candidates have the commitment and skill to 
acquire detailed knowledge about previously unexplored arenas within their own campuses. 
Principal preparation programs can leverage their efforts in this quest by providing hands-on 
experiences to candidates that teach authentic, team-based research skills. These learning 
experiences will be far more effective if they occur in a setting that is both familiar and 
important to the learner. 



A Training Model 



A combination of needs came together to create a training model that enabled 
educational leadership candidates to evaluate the principal preparation program within the 
Department of Educational Leadership and Counseling at Prairie View A&M University. The 
department had not undergone an extensive internal program review until its involvement in 
the Texas Principals’ Lighthouse Initiative, a consortium of approximately 15 universities in 
Texas committed to the continuous improvement in the delivery of principal preparation in 
the state. Therefore there was an urgent need for data to be collected and analyzed in a short 
time frame. 

An educational administration research class needed a client for a program 
evaluation project that would involve contracting with a department within the university or a 
school district. The purpose would be to enable these future and current administrators to 
design and conduct an evaluation study from beginning to end — a turnkey operation — within 
an authentic environment. Ultimately these candidates would present their findings and 
recommendations to their client. Ironically, the academic department in which they would be 
earning a master’s degree also became their client. 

During the first week of the 15-week inquiry candidates conducted a review of 
related literature on principal preparation program evaluation, adult learning methods and 
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standards promulgated by NCATE, the National Policy Board and the State Board of 
Educator Certification (Texas). Candidates interviewed other candidates and faculty members 
to determine what variables and questions would be most useful. Faculty opinions were 
sought on the questionnaire before field-testing was conducted. Sixty (N=60) candidates sat 
for the 54-item survey. Questions were added or deleted or wordings were adjusted to better 
reflect intent of the study. 

Initially, some candidates were quite visceral about a few of the issues. They were 
instructed in the ethical considerations of evaluation research as well as the practical side of 
conducting a study does not threaten nor harm the client. Rather the point was to help a 
willing client see its own deficits and plan to improve its processes and performance. 
Significant professional growth and maturation occurred among the class when they came to 
understand that the evaluation was to guide program improvement and not was not merely an 
outlet for pent-up negative feelings. Role objectivity prevailed and the candidates conducted 
the study with great expectation of changes in the program. 

After the templates had been revised and distributed, the candidates worked in pairs 
to learn how to set up a Statistical Program for the Social Sciences (SPSS) template that 
would be used to record and analyze data. They worked in the same area of a computer lab so 
that consistency and refinement of instructions could be maintained. Each pair of candidates 
was responsible for creating a portion of the template. One of the pairs with more specialized 
experience in data analysis managed the development of the entire template. This team 
maintained version control and documentation as the other teams added a portion of the 
template. A single version of the fifty-four-question survey template was distributed by email 
to each of the pairs of candidates. This master version of the SPSS template was used to 
record data. Any revisions to the instrument and template were noted prior to distribution to 
the teams. 

Training in the use of SPSS continued with each team inputting data from a portion 
of the 60 available field-tested instruments. Each pair determined which individual would 
read out the response and which would perform the data entry. This ensured greater reliability 
and reduced data entry error and fatigue. When all data had been entered, the results of the 
survey were analyzed to yield frequency distributions, summary statistics, correlations and 
various comparisons of means. 

Written comments of respondents were recorded as text in Microsoft word. The data 
input by the various pairs of candidates were combined into a master list. Implications of the 
field-test data were discussed and the implications for practice and policy were addressed. It 
was understood by all participants that the data gathering, organizing, inputting and analysis 
were just for practice. The real study lay ahead when the class participants would attempt to 
survey the entire population of candidates for master's degrees in educational leadership on 
three campuses of Prairie View A&M University. 

Over a two- week period candidates distributed the revised surveys in most classes of 
educational leadership. Only one section of a class was not included in the study. By the end 
of the two-week period, because some students attended multiple classes in the Department, 
some classes had only a handful of students who had not completed the survey. The 
population of the final study totaled hundred fifty- four (N=254) candidates in educational 
leadership. Over the next two weeks, the pairs of candidates met during class to input the new 
data applying the same procedures that had been used for the preliminary training data. 
Written comments were coded and a table was constructed to show greater detail for some of 
the concerns indicated in the surveys. 

The following two weeks of the study were spent discussing the findings and 
determining what recommendations they would make to the faculty of the Department. Data 
were distributed to faculty and to candidates who had been most instrumental in conducting 
the study. The candidates felt certain that the study was representative of their own thoughts 
and feelings of those of their peers. They made recommendations with great clarity and 
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conviction. They communicated to the faculty the urgency of making changes so that their 
own experiences and those of other candidates could be improved. 

This study produced a number of surprising findings and some that were expected. 
The surveys confirmed some things that had been known for a long time. For several years 
there had been no systematic approach to gathering and reporting the kinds of data collected 
in this study, the issues were never addressed. Almost immediately after the data from this 
study were reported the Department, the Chair and faculty began moving on some of the most 
pressing issues. One example was in the area of student confusion over registration 
procedures, conflicting information over degree plan options and the need for numerous 
signatures on a variety of papers. The role of advisor had become one of signing registration 
forms, providing access codes for registration and signing waivers for course requirements. 
Papers were frequently lost, professors and students missed connections, and students were 
being sent to a variety of offices before they could find someone to answer their questions. 

When department secretaries were interviewed to find out what had been the most 
frequently asked questions during each work day, they indicated that most of their days were 
spent answering the same questions related registration and degree plans. This corroborated 
the findings in Table I below: 



Table I 

Findings of the Candidate Survey that resulted in immediate corrective action: Stressors 
experienced by candidates (N=256) 



Not Moderately Highly 

Stressful Stressful Stressful 



1 . Departmental procedures and paper work 

2. Lack of signs posted during registration 

3. Lack of available classes at need times 
and locations 



114(44.5%) 
129 (50.4%) 
108 (42.2%) 



100 (39.1%) 
72 (28.1%) 
92 (35.9%) 



42 ( 16.4 %) 

55 (21.5%) 

56 (21.9%) 



After inspecting the numbers included in Table I and the fifty-one remaining 
variables, the candidates conducting the study came away with an appreciation of the power 
of their data and the implications that data had for recommending reform. They also grew to 
appreciate the importance of moving away from the traditional use of numbers by schools and 
toward an approach that was more customer-oriented. 



Seeing things Differently: Fixing Broke Stuff 



A traditional approach to looking at survey data would be to squeeze as much good 
news out of them as possible. In a traditional system, it would be pointed out that the greatest 
number of people in each area of Table I experienced no stress about these issues. The 
conclusion in a traditional approach would be drawn that those who reported either “moderate 
stress” or high “stress” were whiners. In a blame-the-victim maneuver they would be 
chastised for not doing a better job of reading the instructions in the university bulletin or 
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some other more obscure document set out in some other not well-known corner of the 
department. . The result of this approach would be little or no change and complete dismissal 
of the concerns of the relatively smaller numbers of individuals having problems. It is 
axiomatic among those who are familiar with higher education literature, Alexander Astin's 
admonition that when students voices' cannot be heard within the system, they will vote with 
their feet and go elsewhere. 

A more customer-oriented approach that relies on data such as this would take a 
much more critical view of the findings. This approach to management and leadership are 
based on the Six-Sigma principle that candidates on this project became immersed in as a 
theoretical paradigm. Pande, Neumna & Cavanaugh (2000) wrote: “In Six-Sigma, customer 
focus becomes the top priority. For example, the measures of Six Sigma performance begin 
with the customer. Six Sigma improvements are defined by their impact on a customer 
satisfaction and value” (p. 18). They further argue that developing a measurement 
‘infrastructure’ is critical for the organization to monitor and respond to opportunities and 
challenges. 

Applying this notion, candidates conducting the study noted that they or their peers 
do not come to a university, or to any school for that matter, just to get stressed out. They 
have higher purposes in mind and they expect the university to respect this. As future 
administrators they found this more critical approach to data analysis refreshing. They came 
to see this approach to data interpretation as fundamental to their own success as principals. 

For example in Table I, rather than note the comparatively larger percent of people 
who reported no stress, they noted that 55.5 per cent of the candidates surveyed find 
departmental procedures and paperwork either moderately or highly stressful, 49.6 per cent 
found lack of signs in buildings indicating the next step during registration to be moderately 
or highly stressful, and 57.8 per cent of candidates found the lack of available courses at times 
and locations desired to be moderately or highly stressful. Written comments corroborated 
these numbers and provided greater explanation of concerns such as trying to find specific 
advisors to sign forms, forms being lost, having to refile forms multiple times, being sent to 
several offices before someone would begin to address their concerns, being unable to register 
at all due to the fact that so many courses had prerequisites and being offended due to 
rudeness of staff. 

Administration candidates conducting this study learned that a customer-oriented 
approach uses every percentage point in the negative column as an opportunity to improve 
services. It represents an opportunity also to improve results. 

When these and other data were presented to the faculty of the Department of 
Educational Leadership and Counseling there was little surprise. What did change was the 
level of urgency attached to the problem. Within two weeks of presentation of these data, the 
Department went to unrestricted on-line registration for candidates, elimination of 
prerequisites that served little purpose and that created bottlenecks and delays in candidate 
program completion. Data from Multiple forms were simplified and signature requirements 
were eliminated. Contradictions were eliminated. 

All staff members were placed on notice that no candidate was to be sent or 
transferred to another office. Every piece of paper was to be handled by one administrative 
assistant. All registration matters that remained after implementation of on-line registration 
were to be handled by a second administrative assistant. Advisement for professors and 
students was redefined to entail counseling candidates on matters of academic content and 
their career development. Five on-line courses were created and approved for immediately 
launching where none had previously existed. All of this occurred within two weeks of receipt 
of the students' report. 
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Candidate Response 



After the candidates who conducted this study had efficiently collected and analyzed 
the data they were amazed at what had occurred. The completed SPSS tables represented the 
first time many had ever seen the results of a school survey that they had completed either as 
an employee or student. The conclusions and recommendations they generated were 
extensive and will take time for the department to process. However, their amazement will 
continue when they sit down to register for the coming semester. Yes sit down. They will 
find that their efforts have already yielded significant results. They will never again have to 
drive to the university or go through the endless registration paper chase that had become a 
dreaded but inevitable event. Now they will be able to register for the courses from the 
comfort of their homes. 

Every project participant went away as an experienced evaluator. Each also had 
come to feel the power of having their voices not only heard but also amplified through a 
simple data gathering and analysis process. More importantly they learned that when data are 
systematically gathered for a school that has innovative leadership that invites and acts on 
good data, meaningful things could happen. Additional stories about these data and future 
changes will be the subject of future articles. 
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